System For 3D Monitoring And Analysis Of Motion Behavior Of Targets

ABSTRACT

The present invention relates to a system for the 3-D monitoring and analysis of motion-related behavior of test subjects. The system comprises an actual camera, at least one virtual camera, a computer connected to the actual camera and the computer is preferably installed with software capable of capturing the stereo images associated with the 3-D motion-related behavior of test subjects as well as processing these acquired image frames for the 3-D motion parameters of the subjects. The system of the invention comprises hardware components as well as software components. The hardware components preferably comprise a hardware setup or configuration, a hardware-based noise elimination component, an automatic calibration device component, and a lab animal container component. The software components preferably comprise a software-based noise elimination component, a basic calibration component, an extended calibration component, a linear epipolar structure derivation component, a non-linear epipolar structure derivation component, an image segmentation component, an image correspondence detection component, a 3-D motion tracking component, a software-based target identification and tagging component, a 3-D reconstruction component, and a data post-processing component In a particularly preferred embodiment, the actual camera is a digital video camera, the virtual camera is the reflection of the actual camera in a planar reflective mirror. Therefore, the preferred system is a catadioptric stereo computer vision system.

CROSS REFERENCE RELATED APPLICATIONS

This application is a continuation of International Patent ApplicationNo. PCT/US2006/026602 filed on Jul. 7, 2006 which claims priority toU.S. Provisional Patent Application No. 60/697,135 filed Jul. 7, 2005.

BACKGROUND OF THE INVENTION

1. Copyright Notice

A portion of this patent document contains materials, which are subjectto copyright protection. The copyright owner has no objection to thefacsimile reproduction by anyone of the patent document or the patentdisclosure, as it appears in the Patent and Trademark office patent fileor records, but otherwise reserves all copyright rights whatsoever.

2. Field of the Invention

Generally, the present invention relates to the monitoring and analysisof the behavioral phenotype of targets, such as vertebrates (e.g. zebrafish, or Danio rerio, More specifically, one aspect of the invention isdirected to the automatic monitoring and analysis of the 3-Dmotion-related behavior of laboratory animals, such as locomotionactivity, motor activity, home cage behavior, aggression, antipredatory,group preference, and light preference paradigms, under specificbehavioral paradigm experiments, for a individual animal, or a group(school) of animals, in either real-time and on-line mode or off-linemode. The laboratory animals can be genetically altered animals afterknock-in, knock-out, or transgenic operation, or experimental animalsafter being exposed to drugs, chemicals or certain environments.

3. Related Art

Experimental laboratory animals such as wild-type animals, geneticallyaltered (transgenic, knock-in, or knock-out) animals, drug-treatedanimals, and animals after chemical mutagenesis operations, have beenextensively used as human models in various biological, clinical,biomedical, environmental, and military research areas, includinggenomic research, studies of genetic functional disorders, phenotypicdrug screening, toxicology, bio-sensing, and bio-defense, just to name afew. This is due to the fact that humans and animals share extensivegenetic and neuroanatomical homologies, which are widely conserved amongdifferent species. The behavior studies of the animal models areparticularly useful in post-genomic research areas towards identifyingknown genotypes, quantifying the behavior responses induced by variousneurological disorders, and revealing the toxicity and efficacy of drugcandidates.

Deficiencies in motor function may be caused by genetic mutations or bythe effects of chemical compounds. Motion-related animal behavioralstudies are useful in understanding the effects of different genotypeson the development of various motion-related functional diseases, suchas Huntington's disease and Parkinson's disease, as well as the effectsof drugs or any chemical compound on humans. Typical animal models usedfor these purposes include rodents such as mice and rats, vertebratessuch as zebrafish and goldfish, and insects such as drosophilae. Avariety of standardized animal behavioral tests have been designed withthese models. For example, the behavioral tests for rodents are composedof open field, home cage, water maze, and social behavior paradigms,while the behavioral tests for vertebrates are composed of swimminglocomotor activity, antipredatory behavior, and group preferenceparadigms. The key parameters describing the phenotypic behavior havebeen defined for these tests. For example, the swimming locomotivebehavior of zebrafish can be classified and analyzed by travelingdistance, swimming speed, turning angle, average rate of change ofdirection (RCDI), net to gross displacement ratio (NGDR), body waveshape, and tail beat amplitude and frequency, etc.

Among all the standardized behavioral tests for laboratory animals ofvarious species, motion information and spontaneous activity informationare of great importance for phenotypic screening. Such information canbe obtained from open field (for locomotor activity test) and home cageparadigms.

The monitoring of motion patterns of laboratory animals has historicallybeen accomplished by human observation and/or off-line manual countingon pre-recorded videotapes, which inevitably resulted in inaccurate,inadequate, and subjective data and observation results. Furthermore,human observation methods have significant drawbacks such as lacking ofquantitative data, large observation variations, labor-intensiveness,high costs, and missing of information along the depth direction ofhuman eyes. Recently, researchers have developed various computerizedapparatuses and methods to automatically monitor the locomotion/motorbehavior of animals, including photobeam cage, force actoplate, and 2-Dvideo recording combining with off-line video sequence analysis, just tolist a few. See, e.g., S. Kato, et al, A computer image processingsystem for quantification of zebrafish behavior, Journal of NeuroscienceMethods, 134(2004), 1-7; and J. Chraskova, et al., An automatic 3-Dtracking system with a PC and a single TV camera, Journal ofNeuroscience Methods, 88(1999), 195-200. Among these methods, the videorecording method has unique advantages over other methods, such asnon-contact setup, high sampling frequency, high spatial resolution,long monitoring period, the ability of tracking the motion of specificparts of the body, and versatility in tracking the motion of differentspecies. Therefore, the 2-D video recording and analysis method is morewidely applied in the field of animal behavior monitoring and analysis.

However, there are still significant drawbacks associated with theexisting 2-D video monitoring and analysis systems. For example,existing video monitoring systems typically collect those kinematicalparameters describing animal planar motion only by using a single videocamera, i.e., a horizontal plane if the camera views from the top of themotion field. The camera of existing 2-D video systems generally shootsa single view of the animal container thus losing information alongother spatial axes, such as the camera axis perpendicular to the planedefined by the image plane of the camera. Consequently, existing 2-Dvideo systems (e.g., the system described by S. Kato, et al., Journal ofNeuroscience Methods, 134(2004), 1-7) generally can not detect theupward or downward motion of the tested animals, e.g., the rearingmotion of mice and up-and-down swimming motion of zebrafish, because thecamera usually shoots from the top of a mouse cage or fish tank. Inaddition, existing video tracking systems have limited capability inmonitoring multiple moving animals residing in the same container andlose the motion information associated with certain animals if theanimals being tracked are occluded by other animals, or if part of theanimal body, which may be of interest, is occluded by the animal bodyitself. For example, the footpath of mice or rats may be inaccessible tothe 2-D video tracking system if the camera shoots from the top of thearena. In addition, existing video monitoring systems generally do notcorrect for some physical errors or environmental changes, e.g., they donot address the measurement error associated with water refraction andreflection, which should be corrected for, when monitoring fish swimmingmotion.

In general, there has been an increase in demand for automaticphenotypic behavior monitoring systems in the past a few years, whichcan be utilized in various behavior tests of laboratory animals.Examples of automatic systems have been developed according to theseneeds include: photobeam cages, force plate actometers, andanalog/digital video monitoring systems. See, e.g., the articles citedabove. The application of these automatic monitoring systems hassuccessfully solved most of the subjectiveness problems associated withthe conventional methods of human observation, such as low accuracy,labor intensiveness, and the resultant data errors. Among these systemsand methods, the use of a video camera as the motion sensor has providedthe most powerful monitoring capabilities due to the high spatialresolution of the camera and its adeptness to various animal species andenvironment. However, many existing video systems are 2-D in nature andcan only monitor the motion information along two translation axes andone rotation axis, i.e. three degrees of freedom (DOF) defining a planarmotion. Consequently, the gathered time histories of animal motionparameters are incomplete when using these conventional 2-Dvideo-tracking systems. Therefore, a more advanced video system that isable to truly monitor the motion-related behavior of laboratory animalsin 3-D space is demanded.

Real-time 3-D systems have been described wherein two or more camerasare used to capture images. In addition, 3-D systems have been describedthat consist of a combination of a camera and two or more mirroredsurfaces, resulting in non-conventional stereo pairs. See, Gluckman andNayar, A Real-Time Catadioptric Stereo System Using Planar Mirrors, IUW,1998; Lin, J. Yeh, M. Ouhyoung, Extracting 3-D facial animationparameters from multiview video clips, IEEE CGA, 22(6), 2002, 72-80; andJ. Chraskova, et al., An automatic 3-D tracking system with a PC and asingle TV camera, Journal of Neuroscience Methods, 88(1999), 195-200.The systems described by Nayar, Lin and Chraskova relate to the captureof a single stereo image. In their systems, however, systematicimplementations of 3-D video tracking of animal motion are notadequately addressed, such as calibrating the system, dealing withmeasurement error and system noises induced by multiple media, trackingthe motion of a single or multiple animals robustly without aliasing,and tracking multiple animals simultaneously without attaching visiblydistinguishable tags. For example, although reflective mirror isemployed in the 3-D animal tracking system described by Chraskova, themirror is only set at approximate orientation and position while notfurther calibrated for its accurate geometric parameters. In theirsystem implementations for tracking the swimming behavior of fish, themonitoring errors such as water refraction-induced distortion of stereogeometry is not corrected. In the system described by Chraskova,furthermore, a light emitting diode (LED) marker has to be carried byevery animal being tracked, while multiple LED markers have to beactivated in alternate frames (time-sharing regime) in the applicationsof monitoring multiple animals simultaneously. These requirementssignificantly increase both the technical challenge in theimplementation of the behavioral experiments and the uncertainties inthe behavior monitoring results.

Accordingly, there exists a need for improved systems and methods for3-D monitoring and analyzing the motion behavior of one or more testanimals.

SUMMARY OF THE INVENTION

The present invention relates to a system for 3-D monitoring andanalysis of motion-related behavior of test subjects. The systemcomprises an actual camera, at least one virtual camera, and a computer.The computer is preferably connected to the actual camera and installedwith software packages capable of capturing the stereo images associatedwith the 3-D motion-related behavior of test subjects as well asprocessing these acquired image frames for the 3-D motion parameters ofthe test subjects of interest. The system of the invention comprises ofhardware and software components. The d hardware components preferablycomprise a hardware setup or configuration, a hardware-based noiseelimination component, an automatic calibration device component, and alab animal container component. The software components preferablycomprise a software-based noise elimination component, a basiccalibration component, and extended calibration component, a linearepipolar structure derivation component, a non-linear epipolar structurederivation component, an image segmentation component, an imagecorrespondence detection component, a 3-D motion tracking component, asoftware-based target identification and tagging component, a 3-Dreconstruction component, and a data post-processing component.

In a particularly preferred embodiment, the actual camera is a digitalvideo camera and the virtual camera is the reflection of the actualcamera in a planar reflective mirror. Therefore, the system is acatadioptric stereo computer vision system.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is the schematic of an exemplary 3-D animal behavior monitoringsystem of the invention, in which the monitoring of the swimmingbehavior of fish is used as an example.

FIG. 2 is a schematic of an exemplary embodiment of the 3-D animalbehavior monitoring system of the invention, which can be applied tomonitor the locomotion activities, home cage activities, and footsteppaths of mice or rats.

FIG. 3 shows an exemplary hardware-based calibration device component ofthe invention for the calibration of the stereo system of the invention,with a calibration pattern having multiple colored ends as visibleindicators.

FIG. 4 shows an image of the automatic calibration pattern with multiplevisible indicators in the form of high intensity light emitting diodes(LED), which is in the viewing area associated with the actual camera.

FIG. 5 shows an acquired stereo image of an empty fish tank. The threeregions correspond to the viewing areas of actual and virtual camerasand are displayed in bounding boxes.

FIG. 6 illustrates the calculation method for mirror position which iscalculated from the corresponding images of visible indicator sets andthe obtained normal of the mirror plane.

FIG. 7 shows an exemplary of epipolar structure of the invention formedby the actual camera 0 and two virtual cameras 1 and 2 as shown in FIG.1, where e_(ij) denotes the epipoles formed by cameras i and j, andM_(i) stands for the corresponding image of a visible indicator in viewi.

FIG. 8 illustrates the forward and backward refraction process byemploying ray-tracing method. This process is applied in extended systemcalibration, correspondence detection, and 3-D reconstruction whenmonitoring aquatic animals.

FIG. 9 shows a flowchart of a real-time and on-line process formonitoring the 3-D swimming trajectories of aquatic animals.

FIG. 10 shows an image frame where the test fish are identified, taggedand tracked by applying the corresponding software components of theinvention. Top left: original frame; top right: processed frame withtags assigned to the test fish; Bottom: reconstructed 3-D motiontrajectories of the test fish.

FIG. 11 shows a image frame where two feature points of a test mouse areidentified, tagged and tracked by the corresponding software componentsof the invention. Top left: original frame; top right: processed framewith tags assigned to the feature points on the test mouse; Bottom:reconstructed 3-D motion trajectories of the feature points on the testmouse.

FIG. 12 shows examples of reconstructed 3-D swimming trajectories ofthree goldfish in an alcohol addiction test. The monitoring time is 5minutes, where (A) a control group with no ethanol added; (B) a testgroup with ethanol/water volumetric concentration 0.25%; (C) a testgroup with ethanol/water volumetric concentration 0.5%; (D) a test groupwith ethanol/water volumetric concentration 1.0%; (E) a test group withethanol/water volumetric concentration 1.5%.

FIG. 13 shows another example of reconstructed 3-D swimming trajectoriesof three goldfish in an alcohol addiction test. The monitoring time is 5minutes, where (A)'s a control group with no ethanol added; (B)'s a testgroup with ethanol/water volumetric concentration 0.25%; (C)'s a testgroup with ethanol/water volumetric concentration 0.5%; (D)'s a testgroup with ethanol/water volumetric concentration 1.0%; and (E)'s a testgroup with ethanol/water volumetric concentration 1.5%.

FIG. 14 shows examples of reconstructed 3-D swimming trajectories of onezebrafish in an alcohol addiction test. The monitoring time is 15minutes, where (A)'s control with no ethanol added into the water; (B)'sa test zebrafish with ethanol/water volumetric concentration 0.25%; and(C)'s a test zebrafish with ethanol/water volumetric concentration 0.5%.

FIG. 15 shows examples of reconstructed 3-D motion trajectories of amouse head after abdominal ethanol injection. The monitoring time is 10minutes. (A) & (D) are control mice with no ethanol injection; (B) & (E)are test mice after ethanol injection with body weight-normalizedethanol concentration 0.2 g/kg; and (C) & (F) are test mice afterethanol injection with body weight-normalized ethanol concentration 2.0g/kg.

FIG. 16 illustrates examples of reconstructed 3-D trajectories of amouse head after abdominal caffeine injection. The monitoring time is 10minutes. (1.A) & (2.A) are control mice with no caffeine injection;(1.B) & (2.B) are test mice after caffeine injection with bodyweight-normalized caffeine concentration of 6.25 g/kg; (1.C) & (2.C) aretest mice after caffeine injection with a body weight-normalizedcaffeine of concentration 12.5 g/kg; (1.D) & (2.D) are test mice aftercaffeine injection with a body weight-normalized caffeine concentrationof 25 g/kg; (1.E) & (2.E) are test mice after caffeine of injection witha body weight-normalized concentration caffeine of 100 g/kg.

FIG. 17 illustrates sample time series of swimming speed and velocitiesof goldfish in an alcohol addiction test. (A) shows the swimming speedin 3-D space; (B) shows the swimming velocity in X direction (Xdirection is shown in FIG. 11); (C) shows the swimming velocity in Ydirection (Y direction is shown in FIG. 11); (D) shows the swimmingvelocity in Z direction (Z direction is shown in FIG. 11); (1)'s acontrol fish; (2)'s a test fish with ethanol/water volumetricconcentration 0.25%; (3)'s a test fish with ethanol/water volumetricconcentration 0.5%; (4)'s a test fish with ethanol/water volumetricconcentration 1.0%; and (5)'s a test fish with ethanol/water volumetricconcentration 1.5%.

FIG. 18 illustrates sample time series of turning speed and velocitiesof goldfish in an alcohol addiction test. (A) shows the turning speed in3-D space; (B) shows the turning velocity about X axis (X axis is shownin FIG. 11); (C) shows the turning velocity about Y axis (Y axis isshown in FIG. 11); (D) shows the turning velocity about Z axis (Z axisis shown in FIG. 11); (1)'s a control fish; (2)'s a test fish withethanol/water volumetric concentration 0.25%; (3)'s a test fish withethanol/water volumetric concentration 0.5%; (4)'s a test fish withethanol/water volumetric concentration 1.0%; and (5)'s a test fish withethanol/water volumetric concentration 1.5%.

FIG. 19 illustrates sample time series of distance and displacementsfrom trajectory center of goldfish in an alcohol addiction test. (A)shows the distance from trajectory center in 3-D space; (B) displacementfrom trajectory center in X direction (X direction is shown in FIG. 11);(C) displacement from trajectory center in Y direction (Y direction isshown in FIG. 11); (D) displacement from trajectory center in Zdirection (Z direction is shown in FIG. 11); (1) a control fish; (2) atest fish with ethanol/water volumetric concentration 0.25%; (3) a testfish with ethanol/water volumetric concentration 0.5%; (4) a test fishwith ethanol/water volumetric concentration 1.0%; (5) a test fish withethanol/water volumetric concentration 1.5%.

FIG. 20 illustrates a sample of behavioral response curves of goldfishin alcohol an addiction test. (A) shows point estimates, confidenceintervals and response curve of the average distance from trajectorycenter versus ethanol concentration; (B) shows point estimates,confidence intervals and response curve of average 3-D linear speedversus ethanol concentration; (C) shows point estimates, confidenceintervals and response curve of average 3-D angular speed versus ethanolconcentration.

FIG. 21 shows the group means and standard deviations of the up-and-downmotion speed of a mouse head, which is obtained from the experiments inexample 2 (ethanol-induced behavior change of mice).

FIG. 22 shows the group means and standard deviations of the 3-Ddistance from trajectory center (DFC) of a mouse head, which is obtainedfrom the experiments in example 2 (ethanol-induced behavior change ofmice).

FIG. 23 shows the group-wised distribution of the height of mouse headswhen mice are injected with ethanol solutions with different dosages,which is obtained from the experiments in example 2 (ethanol-inducedbehavior change of mice).

FIG. 24 shows the effect of ethanol injection on the rearing behavior ofmice, which is obtained from the experiments in example 2(ethanol-induced behavior change of mice).

FIG. 25 shows the group means and standard deviations of timepercentage, in which mice stay at the rear-half of an animal container.This is obtained from the experiments in example 2 (ethanol-inducedbehavior change of mice).

FIG. 26 shows the group means and standard deviations of the timepercentage when mice stay at the corners of the animal container. Thisis obtained from the experiments in example 2 (ethanol-induced behaviorchange of mice).

FIG. 27 is an exemplary graphical user interface (GUI) of the inventionfor the on-line and real-time monitoring process for rodent nocturnalbehavior. Blacklights are used as an ambient illumination source whilethe feature points on the rodent ears are painted with fluorescent dyes.

FIG. 28 is an exemplary hardware assembly of the 3-D behavior monitoringsystem of the invention, in which the outer covers are taken off fromthe frames in order to show the arrangement of internal hardware parts.

FIG. 29 is a table of exemplary fundamental kinematical attributes thatcan be extracted from the motion trajectories monitored by 2-D and 3-Dvideo systems, where d, v, a stand for linear displacement, velocity andacceleration, while φ, ω, and α stand for angular displacement, velocityand acceleration, respectively. In a 2-D system, the coordinate systemcan only be transformed freely on the camera image plane with a setcamera axis Z, while there is no restriction in coordinatetransformation in the 3-D system.

FIG. 30 is a table of exemplary motion parameters in both time domainand frequency domain, which are used for further statistical analysisfor behavior end-points. These series are derived automatically from thereconstructed 3-D motion trajectories by data post-processing component.In this table, {X, Y, Z} stands for any Cartesian coordinates in 3-Dspace, which can be attached or transformed according to differentneeds.

FIG. 31 is a table of the division of groups of Example 1 (alcoholaddiction tests on goldfish), wherein the change of 3-D swimminglocomotion of goldfish induced by adding ethanol into water is monitoredand analyzed.

FIG. 32 is a table showing the one-way analysis of variance (ANOVA)results of the average distance-from-center, 3-D swimming speed, and 3-Dturning speed in the alcohol addiction tests on goldfish of Example 1 a“*” denotes a significant difference in the associated motion parameterbetween the groups being compared.

FIG. 33 is a table showing the results from post-hoc mean comparison ofthe average distance-from-center in the alcohol addiction tests ongoldfish of Example 1 a denotes a significant difference in theassociated motion parameter between the groups being compared.

FIG. 34 is a table of the results from a post-hoc mean comparison of theaverage 3-D the alcohol addiction tests on goldfish of Example 1 a “*”denotes a significant difference in the associated motion parameterbetween the groups being compared.

FIG. 35 is a table of the results of a group-wised mean comparison ofthe up-and-down speed of a mouse head, from Example 2 (ethanol-inducedbehavior change of mice) a “*” denotes a significant difference betweenthe two groups being compared.

FIG. 36 is a table of the results of group-wised mean comparison of the3-D distance from trajectory center of a mouse head from Example 2(ethanol-induced behavior change of mice) a “*” denotes a significantdifference between the two groups being compared.

FIG. 37 is a table of the results of a group-wised mean comparison ofthe time percentage of rearing during monitoring process from Example 2(ethanol-induced behavior change of mice) a “*” denotes a significantdifference between the two groups being compared.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

The system and methods of the present invention can automaticallymonitor the motion of one or more targets of interest (as referred toherein, a “target” is any test subject of interest, for example anexperimental laboratory animal of interest or a feature point on thebody of a laboratory animal of interest) in 3-D space and providecorresponding quantitative analyses of key motion parameters in 6 DOF.By using the stereo monitoring system of the invention, target motion istracked and analyzed in an extended 3-D space containing at least sixDOFs instead of in the 2-D space with only three DOFs of conventional2-D video tracking systems. In a preferred embodiment, the system of theinvention comprises an actual camera and at least one virtual camera. Astereo pair may be formed between any two of the cameras, and the systemmay comprise one or more stereo pairs. Actual cameras include, but arenot limited to, a digital video camera, or an analog video camera with aframe grabber digitizer. Virtual cameras include, but are not limitedto, the reflections of the actual camera in reflective planar mirrors,and synchronized cameras that are real cameras which are synchronizedwith the actual camera by software or customized hardware. In apreferred embodiment, the virtual camera is formed by reflecting theactual camera into a planar mirror. The use of planar mirrors eliminatesthe need for synchronization and provides the benefit of low systemcost. In addition, space can be saved by employing planar mirrors forvirtual cameras due to the reflective properties of mirrors. When thevirtual cameras of the system are formed by reflecting the actual camerainto mirrors, the system is a catadioptric stereo system.

In the case of using real cameras as virtual cameras, additionalapproaches are required to synchronize the actual and virtual cameras.Such synchronization is to ensure multiple cameras that form stereopairs to grab the images of same targets at the same time. The camerasmay output more than one image for every frame. In this case, morepixels might be effectively devoted to the images of the targets and thetargets could be view more clearly. Computation time and memory cost mayincrease because multiple images have to be processed in order toreconstruct the 3-D positions of the targets.

In one embodiment the system of the present invention may comprisemultiple cameras (including multiple real cameras and/or multiplevirtual cameras), which can form two or more linearly independent viewsof a target being monitored (i.e., that each contain information whichmay not be present in the other). Preferably the cameras are distributedthroughout space to cover multiple animals' 3-D activities.

In the implementation of the preferred embodiment, virtual cameras areobtained by reflecting the actual camera into the mirrors. In thisembodiment, all the extrinsic parameters of the actual camera aremirrored as the extrinsic parameters of the corresponding virtualcamera, while the intrinsic parameters of the corresponding virtualcamera are kept identical to the actual one. The use of catadioptricstereo cameras is basically equivalent to capturing the images and themirrored images of the targets simultaneously by a single real camera ormultiple real cameras. However, the former method is more systematic tobe implemented as a valid stereo system, and more readily to bebreakdown for modulized organization.

The actual camera and virtual cameras are placed in an enclosed spaceand are configured to gaze at the targets of interest in linearlyindependent directions. Specially designed hardware components areemployed to ensure that consistent image frames are acquired by thecameras during monitoring process, with minimum noise. Such hardwarecomponents comprise a geometric configuration of hardware parts (Is thisdefined anywhere?), a hardware-based noise elimination component, anautomatic calibration device component, and a lab animal containercomponent, which are further described below.

The actual camera is preferably connected to a computer (e.g. a personalcomputer (“PC”)) through an embedded image acquisition card. In thecomputer, in-house developed software packages are installed for thepurpose of the behavior monitoring and analysis. The software packagespreferably include a software-based noise elimination component, asoftware-based calibration component, a synchronization component, anepipolar derivation component, an image segmentation component, an imagecorrespondence detection component, a 3-D motion tracking component, asoftware-based target identification and tagging component, a 3-Dreconstruction component, and a data post-processing component, whichare further described below. These software components may perform suchfunctions as adaptive background generation, image segmentation,foreground-image clustering, foreground-image correspondence detection,target identification and tagging, 3-D motion tracking, 3-D positionreconstruction, smart target re-tagging, automatic measurement-errorcorrection, trajectory-based kinematical analysis, and statisticalanalysis of the kinematical parameters describing the 3-D motion oftargets. In a particularly preferred embodiment, the software comprisescorresponding algorithms capable of analyzing video images and computing3-D motion parameters of targets in real-time and/or from stored imagesequences in an off-line mode.

The system can be configured to enable monitoring and analyzing oftarget behavior in terms of 3-D motion history, which is useful forstudying phenotypic behavior of genetically altered laboratory animals,experimental animals with diseases that result in motor functiondegradation, and/or experimental animals of high through-put drugscreening. In addition, the system can be similarly configured to adaptto various applications, such as animal psychology studies, physiologystudies, neurological studies, and/or 3-D shape re-construction ormorphing of the 3-D shape of a moving object.

The user of the system can create a new monitoring record forpost-processing, which contains the 3-D trajectories of the targets ofinterest. The user can analyze the data by retracing the recorded motionhistory of the targets to produce a 3-D display or can analyze the datain text form. Specific features associated with one or more motionparameters can be extracted and analyzed independently from other motiondata. Classification and statistical tests of the data can beaccomplished with well-known techniques, such as ANOVA (which stands foranalysis of variance) and MANOVA (which stands for multivariate analysisof variance).

System Setup

In a preferred embodiment illustrated in FIG. 1 and FIG. 22, the systemincludes an animal container (tank or cage) accommodating the targets,such as experimental animals, which serves as either an open field or ahome cage/tank; an outer box acting as the environment noise shield, onwhich proper illumination sources are fixed, including ambient lights,UV lights, and/or backlights; one digital video camera shooting towardsone transparent face of the tank, which serves as the actual camera; andone or several reflective planar mirrors, in which the images of theexperimental animals can be seen by the actual camera. Virtual camerasare formed by reflecting the actual camera into each mirror. The actualand virtual cameras gaze at directions different from the axes of eachother. The actual camera captures the video images of a space ofinterest in real time. The video images can be stored in a storagemedium, or can be used to calculate the 3-D motion trajectories of thetargets of interest and then disposed of. Calculating the motiontrajectories in real time has several advantages: first, that thesampling frequency can be increased since no image sequence has to besaved to a computer hard disk, and second, that disk space can be saved,allowing for longer term monitoring.

As noted above, although a digital video camera is used as the actualcamera in the preferred embodiment, other non-limiting actual cameraexamples include analog video cameras with frame grabber/digitizers, andcombinations thereof. In addition to at least one actual camera, thepreferred system also comprises at least one virtual camera. FIG. 1illustrates a preferred embodiment having two virtual cameras, formed byreflective planar mirrors and the actual camera. Other examples ofvirtual cameras include synchronized cameras (which are real camerasthat have been synchronized and may also be calibrated), networkedcameras or reflective curved surfaces. Though, usually thehardware-based and the software-based calibration components in thisembodiment need to be modified to fit other virtual camera options.

The image projected on the image plane of the actual camera may becomprised of multiple areas, including the area bounding the animalcontainer (such as the mouse cage or fish tank), and the areas boundingthe images of the animal container in the mirrors, as shown in FIG. 5.This is equivalent to acquiring the images of the targets by the actualand virtual cameras. By virtue of this configuration, multiple images ofthe moving target can be taken simultaneously from different 3-Dviewpoints, which form stereo correspondence sets that can be used forreconstructing the spatial positions of the targets. In order to achieveappropriate fields of view (FOV) and the best image quality, translationand rotation mechanisms can be applied to the mirrors to enable them totranslate and/or rotate freely in the space. The virtual cameras of theinvention are particularly useful for synchronization between any pairof cameras, including actual cameras, virtual cameras, and combinationsthereof. The image of the target itself and images of the target formedin the virtual cameras are captured by the actual camera, on a singleimage plane; and therefore, multiple actual cameras for capturingmultiple images may be dispensed with. The use of a single image planeand a single actual camera greatly reduces the memory requirement, whichmakes image processing more efficient. The system may comprise as manyvirtual cameras as necessary to provide the desired views and data. Thesystem of the invention is capable of providing detailed images oftargets which can be acquired easily, in a cost-effective manner.

The actual camera is connected to a computer that preferably comprisesstereo computer vision capabilities (i.e., includes software packagesfor stereo image processing and 3-D motion tracking). The image from theactual camera may be fed into the computer at a constant high frame rate(e.g. 15 frames per second (fps) in the preferred system) in a real timemode. The images can be stored in the hard disk for laterreference/analyses or dumped immediately after calculating the motiontrajectories and kinematical parameters associated with the targets ofinterests. Any known image processing software can be used, written in,for example, but not limited to, Visual C++, C#. NET, OpenGL on WindowsXP platform, and other programming languages on other softwaredevelopment platforms.

FIG. 2 illustrates another embodiment of a system of the invention inwhich the mirrors are placed in the desired positions and orientationsto form the images of the feet of the monitored animals, which can becaptured by the actual camera 201 without occlusion. Accordingly, thefootpaths of the monitored animal(s) can be recorded or analyzed inreal-time.

Removal of System Noise

System noise (such as inconsistent or time-varying background induced bythe variation in ambient illuminating conditions, segmentation errorcaused by insufficient contrast between the foreground and thebackground in the acquired images, by the water disturbance whenmonitoring the swimming behavior of fish, or by the reflective images ofthe animals formed on the interfaces of different media, etc) bringsforward significant challenges in monitoring processes becauseirrelevant regions may be mistaken as foreground images of the targets.The erroneous results could consequently lead to incorrect imagecorrespondence, target identification and tagging, and errors in thereconstructed 3-D positions of the targets being tracked. Therefore, thepreferred disclosed system may further comprise hardware-based andsoftware-based noise elimination components. The hardware noiseelimination components allow the system to avoid environmental noisesinduced by inconsistent illumination conditions and the monitoringnoises induced by the formation of reflective images of the targets onthe medium interferences. The software noise elimination components canupdate the image background adaptively to eliminate illumination noiseand filter out falsely segmented target images by a variety of filters,including intensity filters, color space filters, and pixel sizefilters. The software noise elimination components further include thespatial consistency constraints imposed by epipolar geometry and thetemporal consistency constraints imposed by 3-D Kalman filters. Thesegmented images corresponding to possible targets are eliminated asmeasurement noises if they do not satisfy both constraintssimultaneously.

As a component for hardware noise elimination, the cameras and animalcontainer are placed inside an enclosed space (outer box shown inFIG. 1) and sheltered from the environmental illumination conditionsoutside. Consistent light sources are included inside the outer box toprovide consistent illumination of the test animals. By employing thishardware configuration, the environmental noise induced by change ofillumination conditions can be effectively eliminated.

In one embodiment, the hardware noise elimination components alsoinclude the capability of image contrast enhancement. For example, thehardware-based noise elimination components may comprise a number ofuniform white light sources, which are preferably placed opposite to thecameras (both actual and virtual), as shown in FIG. 5. Therefore, smalldisturbances in the lighting conditions become inconsequential in theconsistency of image frames.

In addition, the hardware-based noise elimination components furthercomprise unique system setup and part configurations to avoid theformation of reflective images of targets on the medium interfaces. Thetwo walls of the animal container facing the video camera and mirror 2(see FIG. 1 and FIG. 5) are preferably clear and transparent. They serveas the observation windows for the actual camera and virtual camera 2,respectively, with their normals roughly coinciding with the axes ofthese two cameras. On the other hand, the inner surfaces of other threewalls of the animal container (the two walls opposite to the observationwindows and the wall on the bottom) are surface-machined to be roughuntil no reflective image would form on them. In one embodiment ofmonitoring the swimming motion of fish, these fogged panes are lit bythe light sources as shown in FIG. 5 for contrast enhancement. In thisembodiment, nearly saturated white color can be achieved as thebackground in the image. Foreground targets are thus stressed as darkregions.

In a preferred embodiment, for eliminating reflective images of targetson medium interfaces, the axis of the actual camera is on the plane ofone monitoring window of the animal container and one mirror isconfigured such that the axis of a corresponding virtual camera is onthe plane of another monitoring window. Accordingly, the system isconfigured with the observation window(s) being either parallel to theimage plane of its associated camera or perpendicular to other cameras,to prevent the formation of reflective images of the targets on theobservation windows. This may be achieved by manually or automaticallyadjusting the planar mirrors. To automatically adjust the mirrors forthe optimum configurations of virtual cameras, an algorithm takes theprojections of two monitoring windows of the container as input andaccomplishes the adjustment by a step-wise rectification of thepositions of the virtual cameras until the projection of the twomonitoring windows is in the form as the above.

As a part of software noise elimination component, the background image,in which the targets reside in, can be adaptively updated whenbackground subtraction is performed for target segmentation. Theprevious background image is replaced by an updated image whenever thenoise level becomes larger than a set threshold in the previousbackground image. The ability of the software noise eliminationcomponent to be dynamically updated makes the monitoring more robust andadaptive in common lab lighting environments, under enhanced lightingconditions, or in a natural environment.

In another preferred embodiment when color space segmentation isperformed for target segmentation, the color space attributes (such ashue, saturation and value) of the pixels in the acquired frame arefiltered by color space filter. The pixels with color-space attributesoutside the color space confined by set ranges of values are filteredout. The set ranges of color values correspond to the color attributesof the targets to be monitored.

The software-based noise elimination components further comprise a pixelintensity filter and a pixel size filter to eliminate erroneouslysegmented foreground images of targets. In a preferred embodiment byapplying background subtraction for target segmentation, the pixelintensity filter works by removing the foreground image pixels withintensity value falling outside a set intensity range, which correspondsto the intensity of target images. After target segmentation throughbackground subtraction or color segmentation, the pixels of thesegmented foreground images of possible targets are clustered intomarkers. The marker pixel-size filter is then applied to these markers.Markers with pixel sizes falling outside the set size range of desiredpixel number of target images are filtered out as measurement noise.

Calibration of the Stereo Pairs

The system of the invention preferably comprises a hardware-basedcalibration component and a software-based calibration component. FIGS.3 illustrates an exemplary calibration component in which an auxiliarycalibration apparatus 302 is captured by all the actual and virtualcameras. The images of the apparatus in the mirrors 304 and 305 are seenby the actual camera 301, as well as the image directly captured by theactual camera itself. A correspondence is established automaticallyamong these images, from which the positions and orientations of themirrors, the extrinsic parameters of the virtual cameras, andfurthermore the epipolar structures of the stereo pairs can be obtainedby stereopsis calibration process.

The data collection process for system calibration is as follows. Aimage containing multiple views of the animal container with thecalibration apparatus put inside is compared to a image containing thesame views of the animal container with no calibration apparatus puttingin. This is achieved either by stressing the color information of thecalibration apparatus then HSV (hue, saturation, value) segmenting whichresults in difference image with distinguished intensity and color, orby stressing the intensity information of the calibration apparatusonly. The stressed pixels form several clusters, and the geometriccentroids of these clusters are computed on the image plane with knowncorrespondence, and later are used to solve the calibration equationsfor the positions and orientations of the mirrors.

There are various embodiments of the hardware-based calibrationcomponents, including the calibration apparatus and of the patternsobtained therefrom. For example, an object with visible indicators suchas colored ends as shown in FIG. 3, or several colored LEDs with certainon-and-off sequential orders as shown in FIG. 4, may be used. Thehardware-based calibration components may further comprise a computerport and a cable connecting the computer port and the visibleindicators, which are used to automatically turn the visible indicatorson and off following a preferred sequence by sending out TTL pulses fromthe computer.

Preferred software calibration components include software for intrinsicparameter calibration of the actual camera, calibration of the mirror,calibration of the virtual cameras, and calibration for mediuminterfaces. In a preferred embodiment, these calibration steps areperformed automatically. Although special software packages (theprocessing in which is described below) for calibration have beendeveloped in a preferred embodiment, any general-purpose calibrationsoftware could be applied for intrinsic parameter calibration of theactual camera. The intrinsic parameters of the actual camera can becalibrated according to the standard camera calibration procedures usingthe hardware-based calibration components of the invention. Preferably,a basic calibration procedure including mirror calibration and virtualcamera calibration, and an extended calibration procedure including thecalibration of medium interfaces for the distortion of stereo geometryby water refraction, are achieved with a variety of software-basedcalibration components that applies the stepwise calibration methodsdescribed further below.

First, a software-based calibration component is capable of extractingthe pixels corresponding to the visible indicators (such as LEDs orcolored ends) on a calibration apparatus by differencing two or moreimages (between the background image and any other image with visibleindicators on) to determine the correspondences among the images ofindicators, which are taken by all the cameras. Such correspondence canbe set, for example, through automatic HSV segmentation, throughautomatic intensity segmentation, or through manual selection.

In a preferred embodiment, the software-based calibration component forthe geometric attributes of the mirrors achieves the mirror calibrationutilizing the following calibration method.

A planar mirror is parameterized by four parameters [u, d], where u is anormalized 3-D vector indicating the orientation (normal) of the mirrorplane, and d is distance from the mirror plane to the origin of thecamera coordinates associated with the actual camera or thecenter-of-projection COP (0,0,0)^(T) of the actual camera. Therefore,any point p on the mirror plane would satisfy:

p·u=d   (1)

The orientation vector u satisfies:

Tu=0   (2)

Where

$T = \begin{bmatrix}{- {f\left( {C_{y}^{1} - M_{y}^{1}} \right)}} & {- {f\left( {C_{x}^{1} - M_{x}^{1}} \right)}} & {{C_{x}^{1}M_{y}^{1}} - {C_{y}^{1}M_{x}^{1}}} \\{- {f\left( {C_{y}^{2} - M_{y}^{2}} \right)}} & {- {f\left( {C_{x}^{2} - M_{x}^{2}} \right)}} & {{C_{x}^{2}M_{y}^{2}} - {C_{y}^{2}M_{x}^{2}}} \\\vdots & \vdots & \vdots \\{- {f\left( {C_{y}^{n - 1} - M_{y}^{n - 1}} \right)}} & {- {f\left( {C_{x}^{n - 1} - M_{x}^{n - 1}} \right)}} & {{C_{x}^{n - 1}M_{y}^{n - 1}} - {C_{y}^{n - 1}M_{x}^{n - 1}}} \\{- {f\left( {C_{y}^{n} - M_{y}^{n}} \right)}} & {- {f\left( {C_{x}^{n} - M_{x}^{n}} \right)}} & {{C_{x}^{n}M_{y}^{n}} - {C_{y}^{n}M_{x}^{n}}}\end{bmatrix}$

Here (C^(i), M^(i)) are corresponding pixel centroids of the images ofthe visible indicators in the region of the actual camera (C^(i)) and inthe region of the mirror (M^(i)), respectively; subscriptions stand forthe image coordinates of the pixel centroids, and f is the focal lengthof the actual camera. Afterwards, the normal u can be computed as theeigenvector corresponding to the smallest eigenvalue of T^(T)T viasingle value decomposition (SVD) (See, Lin, J. Yeh, M. Ouhyoung,Extracting 3-D facial animation parameters from multiview video clips,IEEE CGA, 22(6), 2002).

To calculate the distance d from the COP of actual camera to the mirrorplane, a known length L between two visible indicators (P^(C1) andp^(C2)) on the calibration pattern is required. If the projections ofthese two indicators in the image region associated with the actualcamera are C¹ and C² and their correspondences in the mirror region areM¹ and M². As shown in FIG. 6, the following parametric equations hold:

P ^(C1) =t ^(C1)(C _(x) ¹ ,C _(y) ¹ , f)^(T)

P ^(C2) =t ^(C2)(C _(x) ² ,C _(y) ² , f)^(T)

P ^(M1) =t ^(M1)(M _(x) ¹ ,M _(y) ¹ , f)^(T)

P ^(M2) =t ^(M2)(M _(x) ² ,M _(y) ² , f)^(T)

Here t is the parametric representation of a ray from COP (0,0,0). Sincethese four parameters describing the four sighting rays are correlated,a binary search can be done on the space of t^(C1) based on thereflective properties of a mirror, as the following. We first computet^(C2) by ∥P^(C1)−P^(C2)∥=L at the current iteration value of t^(C1).Afterwards both P^(C1) and p^(C2) are projected on the sighting raysassociated with M¹ and M² along the normal orientation u of the mirrorplane, and resulting in two projects P^(M1) and P^(M2) shown in FIG. 6.After computing t^(M1) and t^(M2) via the line intersection, thedistance between the two projects P^(M1) and P^(M2) is calculated asd(P^(M1), P^(M2))=∥P^(M2)−P^(M1)∥. The recursion process terminates when∥d(P^(M1), P^(M2))−L∥<ε, which is controlled by a single tolerancethreshold ε. The distance d from the COP (0, 0, 0) to the mirror planeis uniquely determined with known P^(C1), P^(C2), P^(M1), and P^(M2).Finally, a least-square solution is obtained within the calculated dvalues since multiple visible indicators are used for mirrorcalibration.

Once mirror calibration is completed, a software-based calibrationcomponent derives the extrinsic parameters (such as the location of thecenter of projection and the camera axis) of the virtual camerasautomatically. The extrinsic parameters of the virtual camera areobtained by reflecting the corresponding extrinsic parameters of theactual camera according to the mirror associated with the virtualcamera. In a preferred embodiment, all the geometric operations such asreflective mirroring are performed in the Cartesian coordinatesassociated with the actual camera. Therefore, the extrinsic parametersof the actual camera are known.

In a preferred embodiment, a software-based calibration component isapplied for the calibration of the extrinsic parameters of the stereopairs. A stereo pair may comprise a actual camera and a virtual cameraor two virtual cameras. As described before, computer-controlled visibleindicators may be placed in the animal container and turned on one byone through sending out TTL pulse signals via the computer parallelport. This introduces massive correspondences. The epipole induced by anactual camera and a virtual camera is therefore computed by thesoftware-based calibration component as the intersection of the linesconnecting the corresponding images of the indicators in the associatedtwo views. The epipoles induced by two virtual cameras is computed viaan optimization process such that the following error function isminimized.

${E\left( {e_{0},e_{1}} \right)} = {\sum\limits_{i}\; {\left( {{\left( {M_{0}^{i} \times e_{0}} \right) \times m} - {\left( {M_{1}^{i} \times e_{1}} \right) \times m}} \right)}}$

Here (e₀, e₁) are the epipoles to be computed through a iterationprocess, m is the projection of the screw axis formed by projecting theintersection line of the image planes of the two virtual cameras ontothe image plane of the actual camera, (M₀, M₁) are the correspondingimages in the regions of virtual cameras 0 and 1 of a same visibleindicator, and i is the index of the visible indicator. The physicalmeaning of the above error function is that the epipolar line connectinge₀ and M₀ should intersect the epipolar line connecting e₁ and M₁ on theprojection of the screw axis of the two virtual cameras, m, when theerror function is approaching zero (See, e.g., J. Gluckman, et al, Areal-time catadioptric stereo system using planar mirrors, Proceedingsof Image Understanding Workshop, (1998)). FIG. 7 shows an example of theepipolar geometry of the stereo system shown in FIG. 1, which iscalibrated out by the software-based calibration component for theextrinsic parameters of stereo pairs.

The basic calibration procedure discussed above is only valid inderiving the epipolar geometry when light travels in a single media(such as in the air when monitoring rodent animals). In the case ofmonitoring motion-related behavior of aquatic animals, epipolarstructures obtained by the basic calibration method disclosed above maynot be enough due to the light refraction at the interface between waterand air. In this case, part of the epipolar lines may become curved andthe projective relation between straight epipolar lines may not hold anylonger. In order to adjust the stereopsis geometry and identify theassociated image correspondence as a sighting ray of travels throughdifferent media, extended calibration is required to accommodate lightrefraction. The new epipolar structure having light passing through twodifferent media is calibrated by using a software-based calibrationcomponent containing extended calibration methods of the invention. Thisextended calibration method is preferably as follows.

In the case of monitoring the behavior of aquatic animals, the tank isfilled with water, thus introducing refraction of camera sighting rays.Consequently, the system should be calibrated in two consecutive steps:(1) basic calibration, in which the conventional epipolar structure ofthe stereo pairs are calibrated and derived following the proceduredescribed before; (2) extended calibration, in which the mediuminterfaces that sighting rays may pass are calibrated for theirlocations and orientations. The extended calibration results in thedetection of the distorted epipolar curves corresponding to therefracted sigh rays through multiple media. During the process, a targetmarker in a region will be projected into a 3-D line segment in theanimal container. And this line segment will be projected back ontoanother region as the epipolar curve. Then during correspondencedetection process, target markers passed through by an epipolar curveare deemed to be the corresponding ones that satisfy the epipolarconstraint. Therefore, the epipolar structure is not linear anymore ifmultiple media present in the paths of camera sighting rays.

In the extended calibration process, the exact locations andorientations of the medium interfaces where refraction occurs need to becalibrated out. The extended calibration is based on ray tracing method.The procedure is composed of two sets of computations as forward andbackward refractions.

According to the system geometry as well as the derived epipolarstructure from basic calibration, initial guesses can be made for theparameters of the interfaces. These parameters include the normal u_(i)as well as the distance from the COP of video camera, d_(i), of the twoobservation windows and the water surface in the tank. Here i (i≦3)stands for the index of these media interfaces.

Suppose a stereo pair is composed of two cameras C₀ and C₁, asillustrated in FIG. 8, where C₀ is the actual camera and C₁ is a virtualcamera obtained from reflecting the actual camera C₀ regarding mirror 1.The forward refraction starts by shooting a sighting ray from COP₀towards the image M₀ of a visible indicator m (the 3-D position of theindicator is unknown), which is on the image plane of C₀. The sightingray COP₀ M₀ is refracted on a point S on interface (u₀, d₀). Byneglecting the thickness of all tank walls due to their relatively smalldimensions (approximately 2.5 mm thick), the refracted ray SP (P is any3-D point on the refracted ray) can be computed following Snell's lawwith known refractive indices of air and water. Backward refraction isthen carried out to find the refractorily projected image P₁ of point Pon the image plane of C₁, which is supposed to be the correspondingimage of M₀ in C₁ provided the parameters of the medium interfaces areaccurate thus P is the actual location of indicator m. Letting P_(i1)denote the refraction point of P on interface (u₁, d₁), P₁ is theintersection point of the image plane of C₁ and refracted ray of rayPP_(i1). Therefore, the task remaining is to find the refraction pointP_(i1), on (u₁, d₁). The refraction point P_(i1) is found by performinga binary search on the intersection line between interface plane (u₁,d₁) and the plane determined by P, the projection of P on (u₁, d₁)(denoted by V), and COP₁. In this step, the distance between COP₁ andthe refracted ray of PP_(i1) is used to guide the search until therefracted ray of PP_(i1) passes exactly through COP₁.

By placing the calibration pattern in water, a set of correspondences<M₀, M₁>_(i) can be obtained, where i stands for the i^(th) indicator.After forward and backward refractions on every M_(0i), a set of P_(1i)can be computed following the methods described above. Afterwards, thefollowing error function is minimized to simultaneously calibrate theparameters of the two interface planes

${E\left( {u_{0},d_{0},u_{1},d_{1}} \right)} = {\sum\limits_{i = 1}^{5}\; {{M_{1i} - P_{1i}}}}$

The above error function depends on all the parameters of both interfaceplanes. The optimization process is carried out in an alternate manneruntil sufficient convergence is met. Newton method is employed in theiteration process to find the convergence point efficiently.

In the implementation of the extended calibration method, virtualcameras are obtained by reflecting the actual camera regarding themirrors as shown in FIG. 8. As mentioned earlier, this implementation isequivalent to viewing the images of the targets themselves as well astheir mirrored images of the targets simultaneously by the actual cameraonly.

Target Segmentation through Background Subtraction and Color SpaceSegmentation

In one embodiment, off-line system calibration is performed through thecalibration components before the animal motion tracking starts,following the basic calibration procedure then optionally the extendedcalibration procedure described above. After off-line calibration iscompleted, the system can be switched into on-line tracking mode whereinvideo images are obtained and fed into the computer with a constantframe rate.

In the preferred embodiment of the invention, target segmentation iscarried out by a software-based target segmentation component. Thetarget segmentation component separates the foreground images associatedwith the targets from the background image through two methods, namelybackground subtraction and color space segmentation. In the preferredembodiments, the former is employed in the tracking of aquatic animals,while the latter is applied in tracking the feature points on rodentanimals. However, these two methods can be applied in tracking all typesof animals, either separately or as a combination thereof.

In tracking the 3-D motion of aquatic animals, the system of the presentinvention preferably generates a background image before the trackingprocess begins. Background generation may be accomplished by eithertaking a background image of the animal container without target inside(as shown in FIG. 5), or by eliminating the pixels associated with themoving animals by differencing multiple frames, and then averaging thecolor values of these frames to obtain an original background image.During the on-line tracking mode, a software-based dynamic backgroundgeneration component can also generate an updated background imagedynamically. This is particularly desired if variations occur in thebackground. In this embodiment, a threshold may be set for a desiredacceptable amount of variation in the current background image. When thevariation exceeds that preset threshold, a new background image can bedynamically determined. The criterion of updating the background imageis as the following:

∥I _(t) −I ₀∥≧(1+τ)∥I ₁−I₀∥

Where I_(t) and I₁ are the frames acquired at time t and the first framewhen the tracking process starts (or the first frame acquired afterbackground re-generation), respectively; I₀ is the current backgroundimage, which is either generated before the tracking process, orre-generated in the middle of the tracking process; and τ is thetolerance. In one embodiment, the threshold is preset to be 5% for everypixel on the image. Accordingly, if the total variation for the newimage is larger than 5% from the original, the images are considereddifferent enough to be re-calculated.

Target segmentation may be achieved by background subtraction or colorspace segmentation, or the combination thereof. The target segmentationfrom background subtraction works by obtaining the foreground imagesthrough differencing the current frame with the current backgroundimage, and then clustering the pixels in the foreground images as themarker images of the targets. In the color space segmentation, thesegmentation of target images is achieved by selecting the image pixelswith their color space attributes (HSV or RGB) falling inside set rangesof desired color space values for foreground images. These foregroundpixels are then clustered into markers.

Target segmentation is performed for all images obtained from the actualand virtual cameras on a single image plane. In the software-basedtarget segmentation component, the following clustering algorithm isapplied on the segmented foreground images. The initially segmentedforeground images are composed of scattered pixels obtained bybackground subtraction or color segmentation. First a foreground imageis taken as input. The image is then scanned by a intensity filterand/or a color space filter to obtain a pixel that satisfies a set ofthreshold values. When such a pixel is found, searching is carried outto tell if any neighboring pixels of the same sort can be found. Ifneighboring pixels are found, they are grouped together as a marker.Otherwise the pixel based on which the searching is carried out isdiscarded. If a marker is formed, the image attributes of this marker iscalculated by averaging the pixel groups in this marker, and then thismarker is segmented as a marker representing a possible target image.This procedure continues until all possible markers on the foregroundimage are found.

In the preferred embodiment, all the markers obtained by segmentationand clustering are then tested by a marker-size filter. Those markerswith their sizes falling outside a set size range are disposed thereofas measurement noises.

Correspondence Detection by Epipolar Constraints

In the preferred embodiment of the invention, the epipolar constraintsare obtained by the derived epipolar structure, which is calibrated bysoftware-based calibration components. In a preferred embodiment, atleast two epipolar structures exist in the system wherein the stereogeometry is undistorted. First, the epipolar structure between a stereopair comprising the actual camera and a virtual camera, or between onetarget marker in the region of an actual camera and another marker ofthe same target in the region of a virtual camera; Second, the epipolarstructure between two virtual cameras, or between target markers in theregion of two virtual cameras. For the first epipolar structure, thesoftware-based epipolar derivation component is capable of computing theepipole e through basic calibration process described before. Thecorresponding marker set in the actual camera region and in the virtualcamera region should satisfy the following epipolar constraint:

(C×M)·e=0

Here C and M stand for the two markers, in the region of the actualcamera and the virtual camera, respectively. The physical meaning of theabove epipolar constraint is that the line connecting the correspondingmarkers C and M should pass through epipole e. For the second epipolarstructure between two virtual cameras, two epipoles e₁ and e₂ can becalibrated out by the epipolar derivation component by the basiccalibration process described before. The corresponding marker set M₁ inthe region of virtual camera 1 and M₂ in the region of virtual camera 2satisfies the following epipolar constraint. The physical meaning ofthis epipolar constraint is that the intersection point of the lineconnecting M₁ and e₁ and the line connecting M₂ and e₂ is located on theprojection of the screw axis, m.

((M ₁ ×e ₁)×(M ₂ ×e ₂))·m=0

In the case of monitoring the behavior of aquatic animals, the abovetraditional epipolar structure becomes insufficient in findingcorresponding markers. Consequently, a different scheme is used to findthe corresponding markers by using epipolar curve. Such epipolar curvesare nothing but the projections of the refracted sighting rays on theimage planes of a stereo pair, which are corresponding. As mentioned inthe extended calibration process, ray tracing method is used in thedetection of such epipolar curve, as the following. Given a marker M₀ ina region, we first project it back in the 3-D space as a sighting ray.Then we obtain the refracted ray segment inside the animal container. Webreak this refracted ray into small line segments. Consequently, we haveseveral end points (the ends of these small line segments) along therefracted ray. By projecting these points to another region viarefraction, we have a piecewise linear epipolar curve in that region.And the corresponding marker Ml will be passed through by the epipolarcurve.

Again, two kinds of refractions are involved in the derivation of theepipolar curve so that the corresponding marker set can be detectedthrough the curved epipolar constraint: forward refraction and backwardrefraction. Forward refraction starts from a target marker and thebackward refraction starts from a 3-D point on the refracted sightingray associated with this marker. The relation between the rays beforeand after refraction on a medium interface can be described as thefollowing according to Snell's law:

$\frac{\sin \; \theta_{0}}{\sin \; \theta_{1}} = \frac{\lambda_{0}}{\lambda_{1}}$

where θ₀ and θ₁ represent the angle between the incident and refractedrays and the interface normal and λ₀ and λ₁ are the refractive indicesof the two media such as air and water. The procedure of the forward andbackward refraction is similar as described in the extended calibrationprocedure (calibration of the stereo pairs).

With slight modification, the ray tracing method used in thecorresponding detection and the derivation of refracted epipolarstructure can also be readily applied for the reconstruction of the 3-Dpositions of targets.

Target Tracking, Identification, Tagging

One embodiment of the present system includes a software-based target3-D motion-tracking component and a target identification/taggingcomponent.

In a preferred embodiment, the software-based 3-D motion-trackingcomponent comprises two types of consistency constraints, including thespatial consistency between different views; and temporal consistencyover image sequence. These two sets of constraints are dynamicallyassigned with adaptive weights and are imposed concurrently during thewhole monitoring process.

The constraints for spatial consistency work as the followings. Whenmarkers correspond to possible targets are obtained through segmentationand clustering, the correspondences between them are identified. Thedetection of such correspondences is realized by the generation ofGeometric Compliant Correspondence Sets (GCCS) interconnecting thecorresponding markers in different views. For every frame, thederivation of GCCS is guided by the conventional epipolar geometry whenno refraction occurs, or by the nonlinear epipolar curve constraintswhen sighting rays pass through more than one medium. The spatialconsistency is enforced by the detection of such GCCS on a per framebasis, which is time-irrelevant.

The constraints for temporal consistency work as the followings. Alinear 3-D Kalman filter is designed to predict the current 3-Dlocations of the targets from the reconstructed locations and velocitiesof these targets in the previous frame. In the 3-D Kalman predictor, astate vector

x_(i)(t)={p_(xi)(t) p_(yi)(t) p_(zi)(t) v_(xi)(t) v_(yi)(t)v_(zi)(t)}^(T) is kept updated over frame sequence for the 3-D locationp_(i)(t) and velocity v_(i)(t) of the i^(th) target. For frame t+1, theprediction of system state {circumflex over (x)}_(i)(t+1) is made basedon the state vector x_(i)(t) in frame t:

{circumflex over (x)}(t+1)=Φx _(i)(t)

Here Φ is a constant state transition matrix. In our system, a 3D linearsystem is employed to predict the new state, as described in thefollowing equations.

p _(i)(t+1)=p _(i)(t)+v _(i)(t)

v _(i)(t+1)=v _(i)(t)

The state transition matrix is as the following.

$\Phi = \begin{bmatrix}1 & 0 & 0 & 1 & 0 & 0 \\0 & 1 & 0 & 0 & 1 & 0 \\0 & 0 & 1 & 0 & 0 & 1 \\0 & 0 & 0 & 1 & 0 & 0 \\0 & 0 & 0 & 0 & 1 & 0 \\0 & 0 & 0 & 0 & 0 & 1\end{bmatrix}$

Together with the estimation of the 3-D locations of the target at framet+1, a predicted state covariance matrix P_(i)(t+1|t) can also beobtained from the Kalman predictor. Therefore, the 2-D projections ofthe estimated 3-D locations are computed through a backward refractionprocedure, while the largest eigenvalues of matrices P_(i)(t+1|t) arealso computed and projected as the radiuses of the search regionsencircling all the projected 2-D markers of the targets.

In a particular embodiment, the target identification/tagging componentis an energy function, which operates preferably according to theprinciple of minimum energy. The energy function takes the segmentedforeground markers of the tracked targets as input. In a particularlypreferred embodiment, the energy function combines two main parameterstogether for target identification. These two parameters are obtained bythe 3-D motion tracking component, including, spatial consistencycomprising a GCCS detected by a variety of epipolar constraints, andtemporal consistency comprising multiple 3-D linear Kalman-filterpredictions with inherited target tags. In order to identify targets andassign proper tags to them on a stepwise basis, iteration is appliedthrough all allowable permutations to find all energy values for all themarker pairs. The pair with the minimum of the energy is considered tobe the correct motion pair inheriting the target tag from the previousframe.

As one of the energy parameter, spatial consistency employs epipolarconstraints obtained by epipolar geometry derivation component. Theseconstraints may follow traditional linear epipolar geometry obtainedthrough basic calibration when single medium presents, or followadjusted nonlinear epipolar structure obtained through extendedcalibration as light refraction occurs. The spatial consistency valuesare tested between possible corresponding markers to yield GCCS, forexample, the value is 0 if the markers do not lye on the epipolar lineor curve, which means that the spatial constraint is not met by themarkers. The value is 1 if the markers perfectly lye on the epipolarline, which means that the spatial constraint is met by the markers.Such spatial consistency value is between 0-1 based on the alignmentbetween markers.

As another energy parameter, temporal consistency employs all the 3-Dlinear Kalman predictors for all the targets being tracked. The targetsare identified by the target identification component, which is based onthe minimization of energy function by combining the spatial consistencyand temporal consistency together. Such overall tracking consistency ismainly achieved by minimizing the energy function shown as thefollowing. Suppose the set of Kalman filter predictions (with inheritedtags) is denoted by N and the detected GCCS is denoted by M, with n tagsand m sets of correspondences, respectively (m≧n due to possible noise),the energy function is:

${E\left( {N,M} \right)} = {\sum\limits_{i,{j = 1}}^{n,m}\; {\sum\limits_{v = 1}^{R}\; \left( {M_{vj} - N_{vi}} \right)}}$

Here i,j and v stand for the i^(th) tag, the j^(th) correspondence ofGCCS, and the v^(th) view, respectively. To find the minimum of theenergy function, literally n×m iterations are required with highcomputational cost. However, the search tree can be heavily pruned byconfining the iterations to the GCCS in the predicted search regions.Both the state vector and the state covariance matrices of the 3-DKalman filters are updated through Kalman correctors after the tags arecorrectly assigned to the GCCS.

It should be noted however that physical target tagging is not required.The system of the present invention is capable of intelligentidentification of the three-dimensional movement of multiple targetsusing algorithms based on spatial consistency and temporal consistency.As used herein, “spatial consistency” means the tendency of the targetmarkers satisfying the epipolar constraints of the stereo system. Asused herein, “temporal consistency” means the tendency of the imageassociated with a target to stay close with the previous image of thesame target in a short time. The system of the present invention doesnot require that the animals be tagged or injected with a trackingdevice, unless special points or feature points are of interest, whichare small potions on the animal bodies.

Object Re-Identification After Visual Collision

The present invention may further comprise a software-based re-taggingcomponent for target re-identification. As two or more animals orfeature points are close enough during a certain time span, theirprojected images may merge together in all views. In this case, therobustness of Kalman filters as temporal trackers would degrade, whichusually results in tagging ambiguity as the paths of these animals orfeature points split out after the time span. This is because there ismore than one tag available for each target to inherit at the moment ofsplitting. The re-tagging component is useful where two or more movingtargets, such as zebra fish, are close to each other or make physicalcontact, such that their tags overlap, i.e., their current 3-D positionsappear identical. The re-tagging component of the invention is capableof re-assigning the tags of the two or more targets after near or actualcollisions.

In the preferred embodiment, the re-tagging component comprises severalmethods to apply secondary constraints for temporal consistency. First,dynamic templates are generated and maintained along the frame sequencefor all the targets being monitored. For any frame, such templatescontain all the corresponding pixels of the tagged target markers on theimage plane. These templates are also directed by 2D Kalman filters andare used to distinguish different targets whenever the images of thetargets merge. Furthermore, the extended volumetric 3-D paths of thebody contours of the targets are consistently checked along the temporalaxis to make sure that they don't intersect with each other aftermerge-and-split. FIGS. 10 and 11 show an acquired image frame, theprocessed image, and the on-line reconstructed 3-D trajectories wherethe markers of goldfish and mice are identified and tagged afterimposing both spatial and temporal constraints.

3-D Reconstruction through Triangulation

The system may also comprise a software-based 3-D reconstructioncomponent to reconstruct the 3-D position of the target from itsmarkers, i.e., to determine the actual 3-D position of the target fromthe images obtained from all the cameras. Triangulation method isemployed in the 3-D reconstruction component. The triangulation processfor obtaining the 3-D position of targets without light refraction iscarried out as described below.

After correct correspondences are established among the marker images ofa target, the triangulation component can calculate the 3-D positionswith the following equations:

(P ^(C) +P ^(M))·u=2d

(P ^(C) −P ^(M))×u=0

Here P^(C) and P^(M) are the 3-D position of the target and the mirroredposition of the same target (see FIG. 6), u is the normal of the mirror,and d is the distance from the COP of the actual camera to the mirrorplane. Both P^(C) and P^(M) can be represented in parametric forms asdescribed in basic calibration, with known correspondence of the markersin the regions of the actual camera and virtual camera (see FIG. 6). Ifthere is no available marker of a target in the region of the actualcamera, which is possibly caused by occlusion, the following equationcan be used to eliminate P^(C) for the solution of P^(M)s in twomirrors. It can also be used to recover the actual 3-D positions of thetarget when the mirrored positions are obtained.

P ^(C) =P ^(M)−2(P ^(M) ·u−d)u

In the case of monitoring the motion behavior of aquatic animals, thetriangulation method is justified to accommodate the refraction ofsighting rays, according to the calibrated locations and orientations ofthe media interfaces.

FIGS. 12 and 13 show the reconstructed 3-D locomotion trajectories ofthree goldfish in alcohol response tests. FIG. 14 shows thereconstructed 3-D locomotion trajectories of a single zebrafish inalcohol response tests. FIGS. 15 shows the reconstructed 3-D motiontrajectories of the head of mouse (feature point is set as the middlepoint of two ears) in alcohol addiction response tests. FIG. 16 showsthe reconstructed 3-D motion trajectories of the head of mouse incaffeine addiction response tests.

Data Post-Processing

The system of the present invention further comprises a datapost-processing component. The data post-processing component is capableof smoothing and de-noising the 3-D trajectories of the targets, as wellas extracting the motion variables of interest automatically from thereconstructed 3-D trajectories of targets. The data post-processingcomponent is further capable of performing descriptive and inferentialstatistical analysis on the extracted motion variables. FIGS. 17 to 19show the extracted swimming speed and velocities, turning speed andturning velocities, and distance and displacements from their trajectorycenter from sample 3-D locomotion trajectories of targets in alcoholaddiction tests on goldfish. FIG. 20 shows the results of statisticalanalysis of sample motion variables in goldfish alcohol addiction tests.

FIG. 9 illustrates a particularly preferred system and method fortracking the 3-D motion of laboratory animals.

On-Line and Off-Line Monitoring Process

In a preferred embodiment, the system can work in three modes: theoff-line calibration mode, on-line or off-line monitoring mode, andon-line/off-line post-processing mode.

In the preferred embodiment, the monitoring process is carried out in anon-line mode. In the online mode, a image frame is acquired at a certaintime instant, and then be processed for the current 3-D positions of thetargets. These 3-D positions are added to the current trajectories,which are smoothed, de-noised, and processed for the motion variables ofinterest dynamically. In the off-line monitoring mode, the monitoringsoftware may be decoupled into several modules, including the videocapturing module, the video processing module, and the data analysismodule. In the video capturing module, the video sequence of monitoringprocess is acquired and stored in certain media, such as internal orexternal hard disk and video tapes, in a compressed image format or rawimage format. A time stamp sequence is also stored, which contains thetemporal information associated with the video sequence. The videosequence can be retrieved at any later time for observations or imageprocessing. The video-processing module takes the recorded videosequence as input, and performs all the image processing tasks for theraw re-constructed 3-D trajectories. The data analysis module takes there-constructed 3-D trajectories and the recorded time stamp sequence asinputs, and performs the data processing tasks such as trajectorysmoothing, trajectory de-noising, kinematics analysis, and statisticalanalysis.

Graphical User Interface

In this embodiment, the graphical user interface (GUI) for real-timebehavior monitoring is composed of three windows: the real-time videocaptured by the actual camera, the foreground markers with taggingindices obtained by the software segmentation and identificationcomponents; and the recovered 3-D swim trajectories of targets, as shownin FIGS. 10 and 11. Interactive explorations can be performed on thiswindow using the 3-D motion reconstruction component of the invention.In a preferred embodiment, the 3-D motion reconstruction componentperforms translation, rotation, and scaling according to the cameracoordinates. FIG. 21 shows an example of GUI for on-line monitoringprocess for rodent nocturnal behavior, in which blacklights are used asambient illumination source while the feature points on the rodent bodyare marked with fluorescent dyes.

In a particularly preferred embodiment, the software components of theinvention are Visual C#.NET or Visual C++ programmed within a windows XPplatform with OpenGL graphical support. Examples of other platformsinclude, but are not limited to, MAC, UNIX, LINUX, etc.

EXAMPLES

The following examples set forth exemplary embodiments of the 3-Dbehavior monitoring system of the invention for monitoring and analyzingthe locomotion and motion-related behavior of experimental laboratoryanimals. The examples are provided for illustration only, and nothingtherein should be taken as any limitation upon the overall scope of theinvention.

Example 1 Monitoring the Change in 3-D Swimming Locomotion of Goldfish(Carassius auratus) Induced by Adding Ethanol in Water

Change in fish locomotion responses to various chemical compounds aremonitored, quantified and analyzed by the system and methods of theinvention. For instance, one of such experiments is the alcoholaddiction response of goldfish. In this exemplary behavior monitoringexperiment, goldfish (Carassius Auratus) of average body length 35˜40 mmwith identical biological conditions were screened and raised foralcohol response. The goldfish were then divided into 5 groups, as shownin Table 3. The fish to be tested were kept in identical and naturalenvironment for several days. Before the monitoring process, they weretaken out and habituated for one hour in the tanks with the specifiedalcohol concentration values. Each monitoring process lasted for 15minutes. To exclude any transient response induced by a novelenvironment, the fish trajectories in the last 5 minutes were analyzedfor behavior end points. To demonstrate the capability of the monitoringsystem of the invention to simultaneously track multiple fish, up tothree fish with same treatments were transferred into the test tank fora monitoring run. Other parameters (e.g., time, temperature, etc.) werecarefully controlled for any variance through the overall experimentprocess. FIGS. 12 and 13 illustrate some examples of the reconstructed3-D trajectories from the experiments conducted, with one randomlyselected monitoring run for each group.

Kinematic analysis was carried out automatically by the system once theexperiments were completed. Based on the monitored 3-D locomotiontrajectories of fish, such analysis was carried out for a variety ofmultivariate time series (MTS) for further feature extraction in bothtime and frequency domains. FIG. 17 to FIG. 19 show examples of thesederived time-series (swimming speed/velocity, turning speed/velocity,and distance/displacement from trajectory center, with the trajectorycenter C_(T) of a fish computed as

${C_{T} = {\frac{1}{T}{\int_{0}^{T}{{M(t)}\ {t}}}}},$

where M(t) is the 3-D location of the fish as a function of time, and Tis the time span of the monitoring process) for some cases selectedrandomly from the experiment. FIG. 20 shows the point estimates and the95% confidence intervals of the mean values of the 3-D distance fromtrajectory center (DFC), 3-D linear speed, and 3D angular speed,respectively, against ethanol/water concentration. The leastsquare-fitted response curves are also shown in FIG. 20, indicating therelation between the behavior end points and the ethanol concentrationvalues. Table 4 shows the computed P values from one-way ANOVA analysisregarding the end points of the 5 groups in Table 3. Furthermore, FIGS.34 and 35 show the computed P values from the group-wised post-hoc meancomparison on the average DFC and 3D linear speed, respectively.

As shown in FIG. 19(A) and FIG. 19(B), the goldfish tended to swimfaster along more spread-out paths when the alcohol concentration waselevated from zero. These trends became more pounced when theconcentration was increased until a certain value (approximately 1.3% inthis case). When the concentration was increased passed this point, thelocomotion of the goldfish again became slower as the swimming paths arerestrained to smaller areas until they eventually stopped moving (allthe fish deceased under 2% alcohol concentration) when the alcoholconcentration becomes too high for them to survive. On the other hand,FIG. 20(C) shows that the turning rate of the goldfish becameconsistently lower when the alcohol concentration was increased fromzero.

These observed phenomena coincide with the common knowledge regardingthe alcohol responses of animals. As illustrated in FIG. 33, significantdifference of average DFC and 3D linear speed was found among goldfishgroups, while no significant difference for average 3-D turning rate wasfound. The results from post-hoc tests of means, as shown in FIGS. 34and 35 indicate that for both average DFC and 3-D swimming speed: (i)there was no significant difference among the control group, the testgroup 1 (ethanol/water concentration 0.25%) and test group 2(ethanol/water concentration 0.5%); (ii) there is no significantdifference between test groups 3 (ethanol/water concentration 1.0%) and4 (ethanol/water concentration 1.5%); and (iii) the difference betweenthe control group, test groups 1 and 2 and test groups 3 and 4 wassignificant. These results show similar trends that have been seen inpublished results regarding the alcohol responses of experimental labfish.

Example 2 Rating Ethanol-Induced Intoxication by Monitoring the Changein 3-d Motion of Mouse Head

In this example, the invented 3-D behavior monitoring system is appliedto quantitatively identify detailed differences in mouse locomotioncaused by ethanol injection-induced intoxication. The focus of theassociated experiments was to discover the details in behavior changethat are totally objective yet hard to detect by human observations or2-D monitoring systems.

Female ICR mice 21-25 grams were obtained from Charles River Laboratory.These mice were housed in lab animal facility for at least one weekbefore the behavioral experiments. All mice were housed in standardmouse cages with food and water provided ad libitum. They weremaintained with a 12:12-h-light-dark cycle with lights on at 9:00AMdaily.

With pilot studies with genetically homogeneous mice and arithmeticrating scales, alcohol dosage were determined to be 0, 0. 1, 0.2, 0.5,1.0 and 2.0 g/kg for the behavioral experiments. Accordingly, test micewere divided into 6 groups with 6 mice in each group. In the morning ofthe experiments, yellow and blue fluorescence dyes were painted on theleft and right ears of every mouse respectively. These fluorescence dyesserve as feature points or targets to be monitored. Ethanol solutionswith concentration 0.2 to 2 g/10 ml in saline (0.9% NaCl) were preparedfresh daily. Every mouse was weighed and then injected with ethanolsolution according to the selected dosage. The mouse was then put in theanimal container in the monitoring system for 10 minutes before behaviormonitoring. The inside of the monitoring chamber was illuminated byblack lights to stimulate the fluorescence dyes. For each monitoring,the video image sequence were recorded for 10 minutes and then stored oncomputer hard disk. The recorded video sequences were retrieved laterfor the 3-D reconstruction of the motion trajectories of the featurepoints and data post-processing. The animal container was cleaned aftereach monitoring process was completed.

Motion parameters are computed from the reconstructed 3-D trajectoriesof the mouse heads. The position of mouse heads was taken as the middlepoint of the two ears of mouse. The parameters that are statisticallyanalyzed included the motion speed, angular speed, acceleration anddistance from the trajectory center (DFC). FIG. 21 shows the group meansand standard deviations of the up-and-down motion speed of mouse head.FIG. 22 shows the group means and standard deviations of the 3-Ddistance from trajectory center (DFC) of mouse heads. To analyze therearing behavior of the test mice, average durations of time span whenmouse heads are at different heights were computed on a group-wisedbasis. The following four sections in height were designated: below 17mm, 17 mm to 25 mm, 25 mm to 45 mm, and above 45 mm, with the lastsection corresponding to the rearing motion of mice. Duration in eachheight section is compared between each dosage group. FIG. 23 shows thedistribution of the height of mouse heads in each group. FIG. 24 showsthe effect of different ethanol dosage on the rearing behavior of mice.The time spent in the rear half of the animal container (hidingtendency) is also calculated and compared among groups. In addition,time spent at the four corners of the animal container is calculated andcompared as the indication of exploration tendency. FIG. 25 shows thegroup means and standard deviations of time percentage, in which micestay at the rear-half of the animal container. FIG. 26 shows the groupmeans and standard deviations of the time percentage when mice stay atthe corners of the animal container.

Subject's t tests are used for the group-wised comparison of thesebehavior end-points. Table 7 shows the results of group-wised meancomparison regarding the up-and-down speed of mouse head, with a “*”denotes a significant difference between the two groups being compared.Table 8 shows the results of group-wise mean comparison regarding the3-D distance from trajectory center of mouse head. Table 9 shows theresults of group-wise mean comparison regarding the time percentage ofrearing during monitoring process.

FIG. 15 shows examples of reconstructed 3-D motion trajectories of mouseheads, in which (A) & (D) are the control mice with no ethanolinjection; (B) & (E) are the test mice after ethanol injection with bodyweight-normalized ethanol concentration 0.2 g/kg; (C) & (F) are the testmice after ethanol injection with body weight-normalized ethanolconcentration 2.0 g/kg.

From the statistical analysis on the motion parameters of mouse heads,eight basic motion parameters (3-D speed, 3-D angular speed, 3-Dacceleration, DFC, the projection of 3-D speed on XY plane, theprojection of 3-D angular speed on XY plane, the speed along Z-axis, theangular speed along z-axis) all yielded significant differences (P<0.05or P<0.001) when compared to the control group (saline only) andsometimes the experimental groups (for example, see Tables 7 and 8).

Height distribution in four height sections showed significantdifference especially for the 2.0 g/kg group (see Table 9).

Time spent in the rear half of the holding box did not show anydifference (see FIG. 25).

Time spent at the four comers was significantly different among severalof the groups (see FIG. 26).

The 3-D behavior monitoring system proved to be an efficient tool todistinguish minor changes in motion-related behavior subject toethanol-induced toxicity. These differences could not be detected by eyeyet they were dose-dependent and reproducible. In the present study,only 6 test animals are included in a test group and statisticalsignificance has been detected in many paired comparisons already. Byemploying the same process with more test animals in each group we couldtest all types of mice with their responses to any chemical, withaccurate results.

1. A three-dimensional monitoring system comprising at least one actualcamera, at least one virtual camera, and a computer linked to the actualcamera, wherein the system monitors and analyzes the motion-relatedbehavior of at least one subject.
 2. A three dimensional monitoringsystem comprising at least one actual camera, at least one virtualcamera, and a computer linked to the actual camera, wherein the systemcomprises an expanded calibration component.
 3. A three-dimensionalmonitoring system comprising at least one actual camera, at least onevirtual camera, and a computer linked to the actual camera, wherein thesystem monitors and analyzes the motion-related behavior of more thanone subject.
 4. The system according to claims 1, 2 or 3 wherein thecomputer is installed with software.
 5. The system according to claims1, 2 or 3, wherein the system monitors and analyzes motion-relatedbehavior of subjects in six degree of freedom, including 3 translationsand 3 rotations.
 6. The system according to claims 1, 2 or 3, whereinthe system is capable of monitoring three-dimensional motion-relatedbehavior of one or more subjects.
 7. The system according to claims 1, 2or 3, wherein the system is capable of monitoring three-dimensionalmotion-related behavior of one or more feature points on one or moresubjects.
 8. The system according to claims 1, 2 or 3, wherein thesystem is capable of performing behavior monitoring in an on-line oroff-line mode.
 9. The system according to claims 1, 2 or 3, wherein theactual camera is selected from the group consisting of a digital videocamera and an analog video camera with a frame grabber.
 10. The systemaccording to claims 1, 2 or 3, wherein the virtual camera is selectedfrom the group consisting of a reflective surface and a camera which issynchronized with the actual camera.
 11. The system according to claims1, 2 or 3, wherein the virtual camera is obtained by reflecting theactual camera into a reflective surface.
 12. The system according toclaims 1, 2 or 3, wherein the system is a catadioptric stereo system.13. The system according to claims 10 or 1 1, wherein the actual cameracaptures the images of the targets and the mirrored images of thetargets in the reflective surface.
 14. The system according to claims 1,2 or 3, further comprising more than one actual camera.
 15. The systemaccording to claims 1, 2 or 3, further comprising more than one virtualcamera.
 16. The system according to claims 1, 2 or 3, wherein the actualcamera is capable of acquiring images on a frame-by-frame basis.
 17. Thesystem according to claims 1, 2 or 3, wherein the camera forms more thanone linearly independent view.
 18. The system according to claims 1, 2or 3, further comprising an outer box confining the camera and an animalcontainer within an enclosed space.
 19. The system according to claims1, 2 or 3, further comprising a visible indicator device selected fromthe group consisting of a light emitting diode (LED) or a colored end.20. The system according to claim 19, wherein the visible indicatordevice further comprises an electronic panel and programmable computerports to automatically turn on and off the visible indicators.
 21. Thesystem according to claims 1, 2 or 3, further comprising an animalcontainer.
 22. The system according to claim 21, wherein the animalcontainer has at least two perpendicular faces which define a first anda second monitoring window.
 23. The system according to claim 22,further comprising opaque faces to prevent the formation of reflectiveimages thereon.
 24. The system according to claim 1, wherein the subjectis selected from the group consisting of rodents, fish, insects, worms,amphibians, reptiles, and mammals.
 25. The system according to claim 22,wherein the axis of the actual camera is on a plane of the firstmonitoring window and the axis of the virtual camera is on a plane ofthe second monitoring window.
 26. The system according to claims 1, 2 or3, wherein the computer comprises a calibration component, a stereoimage processing component, a target tracking and identificationcomponent, and a 3-D reconstruction component.
 27. The system accordingto claims 1, 2 or 3, wherein the computer comprises a noise eliminationcomponent.
 28. The system according to claim 27, wherein the noiseelimination component detects noises induced by a change in illuminationconditions and/or by falsely segmented images.
 29. The system accordingto claim 27, wherein the noise elimination component automaticallyeliminates the error caused by noises.
 30. The system according toclaims 1, 2 or 3, wherein the computer comprises a motion-relatedbehavior analysis component.
 31. The system according to claim 30,wherein the motion-related behavior analysis component performskinematical and statistical analysis from reconstructedthree-dimensional motion trajectories of the targets.
 32. The systemaccording to claim 26, wherein the calibration component furthercomprises a basic calibration component and an extended calibrationcomponent.
 33. The system according to claim 32, wherein the basiccalibration component calibrates reflective surfaces and virtualcameras.
 34. The system according to claim 32, wherein the extendedcalibration component calibrates refraction interfaces.
 35. The systemaccording to claim 32, wherein the extended calibration componentemploys a ray tracing method comprising forward and backwardrefractions.
 36. The system according to claim 26, wherein the stereoimage processing component further comprises an epipolar structurederivation component, an image segmentation component, and an imagecorrespondence detection component.
 37. The system according to claim36, wherein the epipolar structure derivation component derives linearepipolar structure in a single medium.
 38. The system according to claim36, wherein the epipolar structure derivation component derivesnon-linear epipolar structure when light passes through two or moremedia.
 39. The system according to claim 37, wherein the linear epipolarstructure is derived from calibrated parameters of the cameras.
 40. Thesystem according to claim 38, wherein the non-linear epipolar structureis derived from calibrated parameters of cameras and refractioninterfaces.
 41. The system according to claim 36, wherein the imagesegmentation component employs a background subtraction method or acolor space segmentation method, or the combination thereof.
 42. Thesystem according to claim 41, wherein the background subtraction methodautomatically and adaptively updates a background image.
 43. The systemaccording to claim 36, wherein the image correspondence detectioncomponent detects correspondence among images of the targets in allviews.
 44. The system according to claim 36, wherein the imagecorrespondence detection component performs correspondence detection byemploying epipolar geometry constraints.
 45. The system according toclaim 26, wherein the target tracking and identification component iscapable of 3-D motion tracking and target identification.
 46. The systemaccording to claim 45, wherein the 3-D motion tracking detects therelated images of targets in all views and in two consecutive imageframes.
 47. The system according to claim 45, wherein the 3-D motiontracking extracts corresponding image sets obtained from imagecorrespondence detection.
 48. The system according to claim 47, whereinthe 3-D motion tracking further employs a 3-D Kalman predictor-correctormethod to track the target images over two consecutive frames.
 49. Thesystem according to claim 45, wherein the target tracking andidentification component identifies the target images on an acquiredframe and automatically assigns tags to the target images.
 50. Thesystem according to claim 45, wherein the target tracking andidentification component identifies the target images by applyingminimum of energy principle on spatial consistency or temporalconsistency, and/or the combination thereof.
 51. The system according toclaim 50, wherein the spatial consistency is the tendency ofcorresponding target images to satisfy epipolar constraints.
 52. Thesystem according to claim 50, wherein the temporal consistency is thetendency of an image associated with a target to stay close with aprevious image of the same target in a short time.
 53. The systemaccording to claim 45, wherein the target tracking and identificationcomponent is capable of identifying multiple subjects without attachinga physical tag to the subjects.
 54. The system according to claim 45,wherein the target and identification component is capable ofidentifying multiple feature points on subjects without attaching aphysical tag to the feature points.
 55. The system according to claim26, wherein the 3-D reconstruction component reconstructs thethree-dimensional positions of the targets from target images.
 56. Thesystem according to claim 26, wherein the 3-D reconstruction componentreconstructs the three-dimensional positions of the targets from two ormore independent views.
 57. The system according to claim 26, whereinthe 3-D reconstruction component performs three-dimensionalreconstruction by employing a triangulation method.